Many techniques currently exist for determining whether a computer program executing on a computing device is operating in an unexpected manner. Computer programs may behave unexpectedly due to bugs in the program, corruption of the memory being used to execute the program, or malicious modification of the computer program. The existing techniques attempt to thwart these issues by protecting either the memory stack or the memory heap, or by providing wrappers around potentially problematic library functions.
One class of techniques that are included as extensions to the GNU Compiler Collection, available from Free Software Foundation, Inc. of Boston Mass., include StackGuard, by Aaron Grier (first included in GCC 2.7.2.2), and ProPolice, by Hiroaki Etoh and Kunikazu Yoda (first implemented as a patch to GCC 3.x). These extensions protect the memory stack buffer. These methods include a “canary” value when loading a function call on the stack, and verify that the “canary” value is not corrupted when the function call is unloaded from the stack. These methods, however, tend to be inefficient and can only thwart attacks that overflow stack buffers.
A second class of techniques that protect the memory heap include the ContraPolice extension for the GNU C Library, by Andreas Krennmair. The GNU C Library (“libc”) is available from Free Software Foundation, Inc. of Boston, Mass. These techniques place random “decoys” before and after heap-allocated buffers and occasionally check the decoys' validity. If one of the checks fails, the program's normal operation can be halted. These techniques, however, also tend to be inefficient and can only thwart attacks that overflow heap buffers.
A third class of techniques, including Libsafe, available from Avaya, Inc. of Basking Ridge, N.J., and the HEALERS approach, by Zhen Xiao, replace functions available in a library such as, for example, libc, with wrapper functions. Some functions in libc, such as, for example, strcpy( ), are known to be unsafe if they are misused. These techniques replace the library functions. These replacement functions, however, will not mitigate issues related to the functions that are not replaced.
A fourth class of techniques that prevent the insertion of executable code includes the WAX methods in OpenBSD, by Theo de Raadt. OpenBSD is an operating system available from The OpenBSD Foundation of Alberta, Canada. These techniques require the operating system to communicate with hardware to determine which memory pages are writable and which are executable, and disallow any page from being simultaneously writable and executable.
A fifth class of techniques makes it more difficult for an attacker to know where code and data will be located in memory. These techniques include, for example, address-space randomization, available in the PaX extension for Linux operating systems at http://pax.grsecurity.net.
A sixth class of techniques, available from Microsoft Corporation of Redmond, Wash., encrypts pointers to decrease the likelihood of a computer program misbehaving due to pointer corruption.